Feat/duckdb backend by gkennos · Pull Request #28 · AustralianCancerDataNetwork/groundworkers

gkennos · 2026-11-06T10:41:15Z

Summary

Adds read-only serving of prebuilt DuckDB vocabulary/graph files and separate DuckDBVSS embedding stores through groundworkers[duckdb].

Configuration and setup support the two-file topology, reject missing or colocated files, and prevent maintenance writes against served assets. Vocabulary lookup, normalized search, ancestry, relationships and embedding retrieval remain available; PostgreSQL full-text search is reported as unavailable on DuckDB.

Graph preparation now uses omop-graph’s packaged classification defaults. Removes Groundworkers’ obsolete CSV copies while preserving explicit custom-directory overrides.

Requires oa-configurator ≥2.1, omop-alchemy ≥3, omop-graph ≥4 and omop-emb ≥3.1. Includes relationship-traversal hardening and integrates backend guidance into the existing documentation.

Production asset building, manifests and deployment validation remain separate follow-on work.

Closes #16

Checklist

Applied exactly one label (breaking, feature, fix, dependencies, or chore)
Tests pass locally (uv run pytest -q)
Lint passes (uv run ruff check .)

Copilot

🟡 Changes recommended

Outstanding defects can over-fetch graph data, discard existing DuckDB schema defaults, accept unusable embedding-store paths, and incorrectly display supported DuckDB stores as unsupported.

4 open issues

What changed in this PR

Adds read-only DuckDB support for prebuilt vocabulary/graph files and separate embedding stores. Graph preparation now uses omop-graph’s packaged classification defaults, addressing #16.

Changes:

Adds DuckDB dependencies, configuration, read-only connection policies, and maintenance guards.
Supports DuckDB normalized search and strengthens bounded relationship traversal.
Adds runtime and packaging tests, setup coverage, and deployment documentation.

File	Description
`uv.lock`	Locks upgraded OMOP packages and DuckDB dependencies.
`tests/unit/test_vector_store_setup.py`	Tests detection of colocated DuckDB file aliases.
`tests/unit/test_relationship_edges.py`	Tests filtered, ordered, bounded relationship queries.
`tests/unit/test_predicate_mapping_classification.py`	Checks classification data from omop-graph.
`tests/unit/test_omop_emb_adapter.py`	Uses approximate floating-point assertions.
`tests/unit/test_normalized_index.py`	Makes test schema bindings explicit.
`tests/unit/test_migration_boundaries.py`	Updates dependency and extra assertions.
`tests/unit/test_mapping_service.py`	Tests traversal budgets, metadata errors, and cache reuse.
`tests/unit/test_mapping_context.py`	Updates delegated argument expectations.
`tests/unit/test_graph_maintenance.py`	Tests upstream defaults and read-only guards.
`tests/unit/test_graph_maintenance_wizard.py`	Tests upstream classification choices and incomplete configuration.
`tests/unit/test_configuration_provider.py`	Adds DuckDB configuration round-trip coverage.
`tests/test_bootstrap_config.py`	Updates mocks for runtime engine creation.
`tests/support/wheel_smoke.py`	Adds installed-wheel DuckDB checks.
`tests/integration/test_omop_duckdb_runtime.py`	Tests file-backed serving, validation, and normalization.
`src/groundworkers/tui/wizards/graph_maintenance.py`	Offers upstream defaults and disables DuckDB preparation.
`src/groundworkers/services/vocab.py`	Adds DuckDB search handling and full-text availability reporting.
`src/groundworkers/services/relationship_neighborhood.py`	Introduces per-seed bounded traversal.
`src/groundworkers/services/mapping.py`	Integrates traversal budgets and strict metadata checks.
`src/groundworkers/services/graph.py`	Exposes bounded relationship queries and strict lookups.
`src/groundworkers/plugins.py`	Explicitly forwards shared-ancestor options.
`src/groundworkers/config/README`	Removes obsolete bundled-data explanation.
`src/groundworkers/config/predicate_classification.csv`	Removes the local classification copy.
`src/groundworkers/bootstrap.py`	Validates embedding files and uses runtime connection policies.
`src/groundworkers/base/sql.py`	Adds DuckDB-compatible normalization expressions.
`src/groundworkers/base/server.py`	Simplifies tool registration and signature assignment.
`src/groundworkers/base/database.py`	Adds DuckDB file validation and read-only engine creation.
`src/groundworkers/application/setup/performance_maintenance.py`	Rejects non-PostgreSQL index maintenance earlier.
`src/groundworkers/application/setup/graph_maintenance.py`	Uses upstream defaults and guards read-only preparation.
`src/groundworkers/application/setup/embedding_setup.py`	Refuses DuckDB store initialization.
`src/groundworkers/application/setup/embedding_population.py`	Validates coverage inputs and blocks DuckDB population.
`src/groundworkers/application/setup/databases.py`	Opens DuckDB readiness connections read-only.
`src/groundworkers/application/setup/configuration_provider.py`	Adds DuckDB setup choices and topology validation.
`src/groundworkers/adapters/omop_graph.py`	Adds bounded active outbound relationship queries.
`pyproject.toml`	Adds the DuckDB extra and raises dependency requirements.
`mkdocs.yaml`	Adds ancestry documentation to navigation.
`docs/usage/installation.md`	Documents DuckDB installation and offline prerequisites.
`docs/usage/configuration.md`	Explains file topology and backend limitations.
`docs/tools/mapping.md`	Links to the runtime ancestry contract.
`docs/from-scratch.md`	Documents prepared-file setup and separate maintenance.
`docs/development/ancestry-mcp-design.md`	Removes the superseded ancestry design document.
`docs/adapters/omop_emb.md`	Documents DuckDB storage and startup validation.
`docs/adapters/ancestry.md`	Documents ancestry semantics and traversal limits.
`config/groundworkers.duckdb.example.toml`	Provides a two-file DuckDB configuration example.
`.github/workflows/ci.yml`	Installs VSS and adds DuckDB wheel smoke checks.

_{Review effort: Balanced}

Copilot

🟡 Changes recommended

Hard-linked embedding paths bypass file-separation checks, allowing initialization to open a writer against the CDM itself, and existing DuckDB stores are incorrectly displayed as unconfigured.

4 open issues

1 previously missed issue

In code that hasn't changed since last review

Reject hard links to the CDM in setup validation

src/groundworkers/application/setup/configuration_provider.py:1308

This setup check also accepts a hard link to the CDM: the resolved filenames differ even though Path.samefile() returns true. The wizard can therefore apply a configuration that violates the separate-file requirement. Check physical file identity for existing paths, while keeping path equality for new targets.

2 resolved since last review

_{Review effort: Balanced}

gkennos added 2 commits November 6, 2026 21:32

Add read-only DuckDB backend support

366a4e1

dependabot

81fa63d

gkennos added the feature label Nov 6, 2026

gkennos requested a balanced review from Copilot November 6, 2026 10:46

Copilot started reviewing on behalf of gkennos November 6, 2026 10:47

View session

Copilot AI reviewed Nov 6, 2026

View reviewed changes

duck db fixes for updated config

d6929f4

gkennos mentioned this pull request Nov 6, 2026

Revert accidental direct push to main (86d759d) #29

Merged

reverting duckdb read-only status for setup commands

ef1c056

gkennos requested a balanced review from Copilot November 6, 2026 12:29

Copilot started reviewing on behalf of gkennos November 6, 2026 12:29

View session

Copilot AI reviewed Nov 6, 2026

View reviewed changes

Comment thread src/groundworkers/base/database.py Resolved

Comment thread src/groundworkers/tui/presenters/embeddings.py Resolved

gkennos added 2 commits November 7, 2026 00:38

use absolute config paths

b4b9b04

fixing relative path in tests

e4d3288

gkennos merged commit 9d4106a into main Nov 6, 2026
5 checks passed

gkennos deleted the feat/duckdb-backend branch November 6, 2026 13:58

Navigation Menu

Repository navigation

Feat/duckdb backend - #28

Feat/duckdb backend#28
gkennos merged 6 commits into
mainfrom
feat/duckdb-backend

gkennos commented Nov 6, 2026

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

gkennos commented Nov 6, 2026

Summary

Checklist

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

🟡 Changes recommended

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

🟡 Changes recommended

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants